Add configurable truncation for string columns by jaykanakiya · Pull Request #19146 · apache/druid

jaykanakiya · 2026-03-12T23:48:01Z

Summary

Adds a configurable maximum string length for string dimension columns. Strings exceeding the limit are truncated during ingestion.

Global config: druid.indexing.formats.maxStringLength
Per-dimension override: maxStringLength field in the dimension spec

Release note

Added a new maxStringLength configuration for string dimensions that truncates values exceeding the specified length during ingestion. Can be set globally via druid.indexing.formats.maxStringLength or per-dimension in the ingestion spec.

Key changed/added classes in this PR

DefaultColumnFormatConfig
StringDimensionSchema
StringDimensionHandler
StringDimensionIndexer

This PR has:

been self-reviewed.
added documentation for new or modified features or behaviors.
a release note entry in the PR description.
added or updated version, license, or notice information in licenses.yaml
added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
added integration tests.
been tested in a test Druid cluster.

clintropolis · 2026-03-13T00:52:44Z

processing/src/main/java/org/apache/druid/data/input/impl/StringDimensionSchema.java

+      @JsonProperty("createBitmapIndex") Boolean createBitmapIndex,
+      @JsonProperty("maxStringLength") @Nullable Integer maxStringLength


drive by comment (i'll have a closer look at rest of PR later)

instead of adding additional arguments here, I was hoping to deprecate these arguments in favor of adding a column format spec similar to was done for auto/json columns in #17762, which could serve as a reference for how this should be wired up. I was planning to move the existing createBitmapIndex and multiValueHandling into such a spec, but just haven't got to it yet. I think this would be much cleaner and less disruptive to call sites going forward. It also allows wiring up to IndexSpec to be able to define job level defaults as a middle place between per column and system wide.

Thanks for taking a look @clintropolis. Adding something like StringCommonFormatColumnFormatSpec would make it cleaner and makes sense to consolidate the configs there. Since it seems like a bigger refactor, does it make sense to do it in a follow up? Let me know what you think.

aho135 · 2026-03-19T00:16:20Z

processing/src/main/java/org/apache/druid/guice/BuiltInTypesModule.java

   */
  private static DimensionSchema.MultiValueHandling STRING_MV_MODE = DimensionSchema.MultiValueHandling.SORTED_ARRAY;
  private static IndexSpec DEFAULT_INDEX_SPEC = IndexSpec.builder().build();
+  private static int MAX_STRING_LENGTH = 0;


Would it make sense to set this to Integer max value? In case this is used elsewhere in the future there wouldn't need to explicit handling for 0 like you have in truncateIfNeeded

We are using NON_DEFAULT to not serialize the default value and I think for integer jackson's default is 0. If we set the MAX_STRING_LENGTH default as int max, it'll serialize this value for each dimension.

aho135 · 2026-03-19T00:19:52Z

processing/src/main/java/org/apache/druid/guice/BuiltInTypesModule.java

  public SideEffectRegisterer initDimensionHandlerAndMvHandlingMode(DefaultColumnFormatConfig formatsConfig)
  {
    setStringMultiValueHandlingModeIfConfigured(formatsConfig.getStringMultiValueHandlingMode());
+    setMaxStringLengthIfConfigured(formatsConfig.getMaxStringLength());


You can take a look at druid.indexing.formats.stringMultiValueHandlingMode in BuiltInTypesModuleTest It would be good to have some test coverage for the new property

Added tests for this property.

aho135 · 2026-03-19T00:21:45Z

processing/src/main/java/org/apache/druid/segment/DefaultColumnFormatConfig.java

    this.stringMultiValueHandlingMode = validateMultiValueHandlingMode(stringMultiValueHandlingMode);
    this.nestedColumnFormatVersion = nestedColumnFormatVersion;
    this.indexSpec = indexSpec;
+    this.maxStringLength = maxStringLength;


Here we should validate that configured maxStringLength > 0, otherwise we can throw an exception or log that we are falling back to the default value

Added validations.

aho135 · 2026-03-19T00:27:21Z

processing/src/test/java/org/apache/druid/segment/StringDimensionIndexerTest.java

+    );
+  }
+
  private long verifyEncodedValues(


Leaving a comment for us to revisit later:

I'm curious what the truncation behavior would be like for MVD

I did have a chat with @abhishekrb19 about multi value strings and we decided to leave out truncation for those.

Ahh okay. I see we're truncating for MVD's with a single value. Would be good to add a test case for that also if that's the intended behavior

jaykanakiya · 2026-03-19T16:44:51Z

processing/src/test/java/org/apache/druid/guice/BuiltInTypesModuleTest.java


-  @AfterEach
-  public void beforeEach()
+  @After


I think there's some mix of junit 4/5 annotations due to which AfterEach was not firing. Updated to use junit 4's annotation for clean up.

aho135

Thanks for the contribution @jaykanakiya! The changes look good to me. I'll let @clintropolis take a pass through as well. I think it would be fine to do the suggested refactoring in a follow up

aho135 · 2026-03-20T17:19:23Z

processing/src/test/java/org/apache/druid/segment/StringDimensionIndexerTest.java

+    );
+  }
+
  private long verifyEncodedValues(


Ahh okay. I see we're truncating for MVD's with a single value. Would be good to add a test case for that also if that's the intended behavior

aho135 · 2026-03-20T17:22:23Z

docs/configuration/index.md

 |`druid.indexer.task.tmpStorageBytesPerTask`|Maximum number of bytes per task to be used to store temporary files on disk. This config is generally intended for internal usage. Attempts to set it are very likely to be overwritten by the TaskRunner that executes the task, so be sure of what you expect to happen before directly adjusting this configuration parameter. The config is documented here primarily to provide an understanding of what it means if/when someone sees that it has been set. A value of -1 disables this limit.  |-1|
 |`druid.indexer.task.allowHadoopTaskExecution`|Conditional dictating if the cluster allows `index_hadoop` tasks to be executed. `index_hadoop` is deprecated, and defaulting to false will force cluster operators to acknowledge the deprecation and consciously opt in to using index_hadoop with the understanding that it will be removed in the future.|false|
 |`druid.indexer.server.maxChatRequests`|Maximum number of concurrent requests served by a task's chat handler. Set to 0 to disable limiting.|0|
+|`druid.indexing.formats.maxStringLength`|Maximum number of characters to store per string dimension value. Longer values are truncated during ingestion. Set to 0 to disable. Can be overridden per-dimension using `maxStringLength` in the [dimension object](../ingestion/ingestion-spec.md#dimension-objects).|0 (no truncation)|


It would be good to mention that truncation does not apply for MVD's

I added the test and doc update in the next commit.

support configurable truncation for string columns

ed7de0a

github-actions bot added Area - Documentation Area - Segment Format and Ser/De labels Mar 12, 2026

clintropolis reviewed Mar 13, 2026

View reviewed changes

jaykanakiya added 2 commits March 12, 2026 22:32

add spelling for checks

9958e34

Merge branch 'master' into string-truncation

9033b75

aho135 reviewed Mar 19, 2026

View reviewed changes

add validations and update tests

4d6bb1f

jaykanakiya commented Mar 19, 2026

View reviewed changes

trigger build

47a0684

aho135 approved these changes Mar 20, 2026

View reviewed changes

aho135 reviewed Mar 20, 2026

View reviewed changes

add single value mvd test and doc update

51b8fe2

		@JsonProperty("createBitmapIndex") Boolean createBitmapIndex,
		@JsonProperty("maxStringLength") @Nullable Integer maxStringLength

Conversation

jaykanakiya commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Release note

Key changed/added classes in this PR

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aho135 Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aho135 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jaykanakiya commented Mar 12, 2026 •

edited

Loading

aho135 Mar 19, 2026 •

edited

Loading